Linear Regularizers Enforce the Strict Saddle Property

Satisfaction of the strict saddle property has become a standard assumption in non-convex optimization, and it ensures that many first-order optimization algorithms will almost always escape points. However, functions exist machine learning do not satisfy this property, such as loss function neural network with at least two hidden layers. First-order methods gradient descent may converge to non-strict points functions, there currently any reliably To address need, we demonstrate regularizing linear term enforces provide justification for only locally, i.e., when norm falls below certain threshold. We analyze bifurcations result from form regularization, then selection rule regularizers depends on an objective function. This is shown guarantee neighborhoods around broad class points, behavior demonstrated numerical examples common literature.